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(57) Abstract 

A bridge for multi-piocessor system includes bus mterfaccs for connection to an I/O bus of a first processing set, an I/O bus of 
a second processing set and a device bus. A bridge contrri mechanism is operable to permit direct memory access to memory of the 
processing sets by a device on the device bus, to arbitrate between the first and the second processing sets for access to the bridge in a 
fiist, split, mode, and to monitor lockstep operation of the first and second processing sets in a second, combined, mode. Tte.dklx RAM 
mechan ism defines a dirtyJndicatoL(c.g.,_a bit) foccarfy)f.a.plurality.otiegions.otpioccs8ing a dirty Indicator being set to a 

predetBrinj ^Taliicwfa ,toJg ^^^ of mentp r y has been written to b y a DMAacc^. One of the processing sets can Oe operaoie m tne 
spilt moSe as a primary processing set to copyl^ content of its memoiy to the other piocessing set(s) and to rccopy regions which become 
identified by the dirty RAM mechanism as having been written to by virtue of the corresponding dirty indication being set In response to a 
synchronization reset operation ftom the primary processing set, on completion of copying the content of the memory regions identified In 
tile dirty RAM mechanism with no ftirther regions having being so identified, tiie bridge can transfer from die split mode to tiie combined 
mode. 
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wo 99/66402 PCT/US99A2429 
TTTLE: TIU^CMNG MEMORY PAGE MODmCATlON IN A BMDGE FOR A MULTI-PROCESSOR 
SYSTEM 

BAOCGROUNn OP THE INVENTION 

nus invention relates to a multi-piocessor computer system including first and second pnxxssing sets 
(each ofwhich may comprise one ormoreprocessors)wluchcommunica^ 

n.c application finds particular appBcation to fault tolerant computer systems where two or more 
processor sets need to communicate with an UO device bus in lockstep with 

enois in order to identify feulty operation of the system as a whole. 

In suchafiuAtDlenmtcomputer system, aaaimis not only to be abte to identify 
a smu^ure which is abk to provide a Mgh degree of system availabiUfy. In order to p^^^ 

availabihty.itwouldbedesirabte for such systems to automaticalty attempt 

AS part of such an automatic recovery process it is necessary to reintegrate the s^ 

toacommonstatusinordertoattemptan^artinkxtoep. An approach to achieving this is ,0 copy the complete 

suue of one of the processing sets (Le. the "good" -e) to the other processing set Hus involves ensunng that the 

c««ent of the memory ofbothprocessoB is the same before tryingarestart in 

However.aprobfcm with the copying of the coBtent of the memory fiomo«e process 

tf^during this time devices connected to the I«)bu»maybemrfdngdhectme^ 

oftheprocessmgsCCs). if a write is m«fc to area of memory which has .heady been copied, this wouW 

mlhememoiy state intheprocessing sets atlheendof the coivnotbcingthe sa^^ 

Ithasbeeaproposedtbemployadirty RAM inaprocessor to h«licate areas of memory whiA 

changedsince the dirty RAM waslastieset A dirty RAM is a bit map having a bit for each block, or page, of 
vAichhitissetwheaawriteacasstotheareaofmcmoryconcen^lismade. However. the p«v«»on of 

adirry RAM in the processing sets wouWnotprovideareliabfcsohrtionto 

ofthei«ocessorbecanseofthediffic»ltiesanddel.ysinacc^ 

An aim of the p^sem invention is to provide a sohmon to the problem of address 

accesses in achieving remstatementofaconcuiTtmstatemfirst and secondpmcc^ 

SUMMARY OP THE INVENTION 

Particular and prefened aspects of the invention are set out in the accompanying independent and 
dependem claims. Contoations of features &om the dependent claims may be « 
independem claims as appropriate and not merely as explicitty set out in th^ 

la accordance with one aspect of the invention, d«re is provided a bridge for a multi-p«^ 

The bridge comprises bus interfiu* for com««tion to an I/O bus of a first proc«^ 

processing set and a device bus. A bridge control mechanism is operable to permit direct memory access to 

^ry of the processing sets by a device on the device bus, to arbitrate between the fim 

processing sets for access ,0 the bridge in a first. spUt. mode, and to monitor lockstep operation of the firs, and 
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second processing sets in a second, combined, mode. A diity RAM mechanism is provided in the bridge for 
monitoring regions of processor set memory modified by direct memory accesses by the device on the dev.ce 

bus. 

An cmbodimcm of die invention is thus able to monitor parts of memory modified by DMA operanons 
5 inioatedby.deviceonthedevicebus. By providing a dirty RAM mechanism in a bridge, this facilitates access 
to the dirty RAM by the processing sets. The reintegration process can involve a mm*er of passes, dtning each 
of which passes dirtied memory is copied ftom a good processing set to a faulty (target) ^ 
During the process of le-integration the good processing set can access the dn^ 

the memory >vhich have been dirtied (in either its own or the target processing set's memory) to be copied on 
10 anypass. 

It should be noted that the bus interfeces referenced above need not be separate components of the 
bridge, but may be incorporated in other components of the bridge, and may indeed be simply comiections for 

the lines of tiic buses concerned. 

in an embodiment of the invention, the dirty RAM mechanism defines a dirty indicator (e.g.. a bit) for 
each of a plurality of regions of processing set memory, a dirty indicator being set to a predetermined value 
when the region of memory has been written to by a DMA access. 

Tte processing sets can be configured such that one of the processing sets is operable in the split mode 
asaprimarypnxxssingsetandtocopythecontemofitsmemorytotheotherpnKessm^ Ifduringthis 
copy openuion some of the regime of the memory are written to byadirect memory access.^ 
of the copy openuion will not be the same in the various processing sets. As a nanih the p 
«H«pies those regions of its memory which have been marked in the dirty RAM 

^tobyvirmeofthecorrespondingdirtyindicatimibeingset nus process can be repeated in a number of 
passes as required. 

In an embodiment of the mvention, the bridge control mechanism comprises an arbiter comiected to 
tfie first and second processor bus inte,fi«« and to the device bus interfece. the art^^ 
operable in the split mode to arbitrate for use of the bridge by the first ami second processing ^ 
on the device bus. Tlie bridge control mechanism is configured to be operable to respond to a synchronise 
reset operation ftom the primary processing set. on completion of copying the content of t^ 
idemified m the dirty RAM mechanism with no finther regions hav^ 

split mode of operation to die cwnbined mode of operation. 

Tl»c dirty RAM mechanism can comprise a dirty RAM configured in random access memory in the 

bridge Altematively.aseparatehaidwarememorydevicemaybeprovidedTtecontemofthedir^ 
be cleared on being read by a processing set Alternatively, two dirty RAMs can be provided, the two dirty 
RAMS being operable in a toggle mode wid. one being written to whik the odKr is being read. Optionally, a 
35 respective dirty RAM could be provided for each processing set 

n«e may be more than two processor bus interfeces for comiection to I/O buses of respective 

processing sets. 

In accordance with another aspect of the invention, there is 

first processing set having an I/O bus, a second processing set having an I/O bus, a device bus. at least one 
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device on the device bus and a bridge as set out above. Each processing set may comprise at least one 
processor, memory and a processing set I/O bus controller. 

In accordance with a further aspect of the invention, there is provided a method of operating a multi- 
processor system as set out above, the method comprismg: 
5 permitting direct memory access to memoiy of the processing sets by the at least one device on the 

device bus; and 

monitoring, in a diity RAM in the bridge, regions of processor set memory written to by the device on 
die device bus. 

A metfiod of le-integration can involve multiple passes of copying areas of memory fiom a first 
10 processing set to a second processing set. the areas to be copied being identified by d^ 

coiTcsponding dirty RAM bit is set 

Tlie le-inlegnition mediod can inchide a set of pn^ 

orlock5tep,mode. 
15 BRIEF DESCRIPTION OF TEIE DRAWINGS 

Excmplaiy embodimcnls of the present invention wiU be described hereinafter, by way of example only, 
wilhrefeience to the accompanying drawings in wh^ 

Figure 1 isaschematicovenicw of afiiulltolcnmta^^ 

20 invention; 

Figuic 2 is a sdwnatic avenricw of a specific impl^^ 

Figure 3 is a sdHanatic rcprcseniatiott of OM implementa^ 

Figure 4 is a sdiematic representation of anoAcr cxairq>le of a processus 

Figure 5 is a scheniatic repiesealaticm of a fardier processing set; 

25 Fignre6isaschcmaticblockdiagiamofananbodimcmofahri^ 

Figure 7 is a schematic block diagram of storage for die bridge of Figure 6; 

Figure 8 is a schematic block diagram of control logic of the bridge of Figure 6; 

Figure 9 is a schematic representation of a routing matrix of the bridge of F 

Figure 10 is anexanq>le implemcntatioiiof Ac bridge of Figure 6; 
30 Figure 11 is a Slate diagram ilhistratingopcrati0nal5ta^ 

Figure 12 is a £k>w diagram ilhistrating stages in dtt operatioa of bridge 

Figure 13 is a detail of a stage of operation fiom Figure 12; 

Figure 14 ilhistrates the posting of I/O cycles in die system of Figure 1; 

Figure 15 ilhistratcs Ac data stored in a posted write buffer; 
35 Figure 16 is a sdjcmatic representation of a slot response register. 

Figure 17 ilhistrates a dissimilar data write stage; 

Figure 18 ilhistrates a modification to Figure 17; 

Figure 19 illustrates a dissnnilar data read stage; 

Figure 20 ilhistrates an alternative dissimilar data read stage; 
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Figure21 isaflowdiagnimsuniiiiarising^ 

Figure 22 is a schematic block diagram explaining arbitration within the system of Figure 1 ; 
Figure 23 is a state diagram ilhistrating the operation of a device bus arbiter 
Figure 24 is a state diagram iUustrating the operation of a bridge arbiter. 
Figure 25 is a timing diagram for PCI signals; 

Figure 26 is a schematic diagram iUustrating the operation of the bridge 



access; 
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Figure 27 isaflow diagram flhuanuh^adkectniemaiy^ 

Figure 28 is a flow diagram of a re^iniegiation piocess including the monitoring of a dirty RAM. 
DESCRIPTION OF THE PREFERRED EM BODIMENTS 

Figure 1 is a schematic overview of a feult tolerant computing system 10 comprising a phmdity of 
CPUsets (processing sets) 14 and 16 and a bridge 12. As shown in Figuie 1. there are two processing sets 14 
and 16 although in other embodiments there may be three or more processing sets. The bridge 12 forms an 
interfece between the processing sets and I/O devices such as devices 28. 29. 30. 31 and 32. In this document, 
d« term •l.rocessing set" is used to denote a group of one or more processors. possiT,^^ 
which output ««1 receive common outputs and inputs. I, should be noted that the alternative term mentioned 
above. 'OUset". could be used instead, and tfvtt these teBB couM be used inteni^ 
document Also, it should be noted that the term -n^idge" is used to denote 
arrangement suitable to interconnecting two or more buses of Ae same or dififemit types. 

Tlie first processing set 1 4 is competed to the bridge 1 2 via a first processing set I/O bus (PA b«^ 
in d« present instance a Peripheral Component Intercom«ct (PO) bus. The second processing set 16 is 
connected to the bridge 12 via a second processing set I/O bus (PB bus) 26 of the same type as the 
(i chereaPabus). n« I/O devices are competed to the bridge 12 via a device I/O bus (D bus) 22. m the 

present instance also a PCI bus. 

AM«,ugh. in the particular example descnl^l. the buses 22. 24 and 26 are aU pa buses^ 

by way of example, and in other embodiments od« bus protocol, may be used and the D-bus 22 

different protocol ftom that of the PA bus and the PB bus (P buses) 24 and 26. 

m processing set, 14 and 16 and the bridge 12 arc operabfc in synchronism mider the contr^^ 

common clock 20. which is comiccled thereto by dock signal line, 21. 

Some of the devices inchuling an Ethernet (E-NET) mterfiice 28 and a SmaU Computer System 
Interface (SCSD interface 29 are permanently competed to the device bu, ^ 

devices 30 31 and 32 can be hot insertable into individual switched slots 33. 34 and 35. Dynamic field effect 
aansistor (FEH switching can be provided for the slots 33. 34 and 35 to en*^ 

such as devices 30. 31 and 32. The provision of Ae FETs enable, an increase in the length of the D bus 22 as 
only those devices which are active are switched on, red^nng the effective total b^ It wdl be 

appreciated that the number of I/O devices wWch may be competed to the D bus 22, and the num^ 
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provided for then., can be adjusted according to a particular implementation in accordance with specific design 
requirements. 

Figure 2 is a schematic overview of a particular implementation of a fault tolerant computer employing 
a bridge structure of the type illustrated in Figure 1 . In Figure 2, the feult tolerant computer system includes a 
plurality (here four) of bridges 12 on first and second I/O mod«iboards (MB 40 and MB 42) order to increase 
the number of I/O devices which may be com««ed and also to improve reliabiUty and redundancy. IHus, m 
Ae embodiment shown in Figure 2. two pix,cessing sets 14 and 16 are each provided on a respective processmg 
set board 44 and 46. with the processing set boards 44 and 46 'bridging- the I/O mod«bo«^ 
42 A first, master clock source 20A is mounted on the first motherboard 40 and a second, slave clock source 
20B is mounted on the second motherboard 42. Oock signals are supplied to the processing set boards 44 and 
46 via respective connections (not shown in Figure 2). 

Finrt and second bridges 12.1 and 12.2 are mounted on the first I/O motherboard 40. TTie first bridge 
,2 1 is comiected to the processing sets 14 and 16 by P buses 24.1 and 26.1. respectively. Similariy. the second 
bridge 12.2 is comieced to the processing sets 14 and 16 by P buses 24.2 and 26.2. respectively. The brrdge 
12.1 is comiected to an I/O daiabus (D bus) 22.1 and d« bridge 12.2 is connected to an I/O databu^ 

" ^ Ttod and fourth bridges 123 and 12.4 are mounted on the second I/O motherix«rd 42. The bridge 

12 3 is connected to the processmg sets 14 and 16 by P buses 24.3 and 263. respectively. Similarly. 4e bridge 
4 is competed to the processmg sets 14 and 16 by P buses 24.4 and 26.4. respectively. TTw bridge 123 .s 
connected to «i I/O databus(Dbus)223 and Ihebridge 1^4 is competed to an Irt5databus(Dbus^ 

It can be seen thattheainmgementshowninFigure2canenablealarge number of I/O devices to be 
connected to the two processing sets 14 and 16 via the D buses 22.1. 22.2. 223 and 22.4 

the range of I/O devices available, or providing a higher degree of redundancy, or both. 

Figure 3 is a schematic overview of one possible configuration of a processing set, such as the 
processingsetMofFigmel. n« processing set 16 could have the same configuration. InF.gure3.apl»ndity 
ofprocesson(herefour)52areconnectedbyoneormoreb«sesMtoaprocessingsetbuscont^ As 
shown in Figures, one or more processing set output buses 24 are comuxted to d« process 
50 each processing set output bus 24 being com«:ted to a respective bridge 12. For example, m the 
arrangemem of Figure 1, only one pRH«sh« set UO bus (P bus) 24 would be provided, w^ 
arrangeme«tofFigurc2.fo«rsachprocessingse.I«)bases(Pb«ses)24w«ddbeprovided In the processmg 
set 14 shown in Figure 3, individual processors operate using the common memory 5 

provide outputs on the common P bus(es) 24. 

Figure 4 is an ahexnative configuration of a processing set. such as the processing set 14 of Fi 

Here a pluiaUty of processor/memory groups 61 are competed to a common internal bus 64. Each 
processor/memory group 61 includes one or more processors 62 and associated memory 66 com^ 
uuemal group bus 63. An interfi«« 65 com«cts the intenuil group bus 63 to the common internal bus 64. 
Accordingly, in the arrangement shown in Figure 4. individual processing groups, with each of the processors 
62 and associated memory 66 are connected viaacommon internal bus 64 toaprocessing set bu^ 
The interfi«:es 65 enable a processor 62 of one processing group to operate not only on th^ 
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„^ 66. but also in memory of ano4er processing group 61 wiflun Ihe processing set 14. The 
processing set bus controller 60 pmvides a common interfiure between the common in.^ 

p^cessing set I/O bus(es) (P b«s(es)) 24 connected U, .be bridge(s) 12. It should be noted tbat although only 
two processing groups 61 an; shown in Figure 4. it wiU be appreciated that such a s^ 

5 number of processing groups. 

Figure 5 illustrates an alternative configumtion of a processing set. such as the processing set 14 of 
Figure 1 Here a simple processing set inctades a single processor 72 and associated memory 76 connected via 
a common bus 74 to a processing set bus controller 70. The processing set bus controller 70 provides an 
inter&ce between the internal bus 74 and Ae processing set I/O bus(es) (P bus(es)) 24 for connection to the 

10 bridge(s) 12. 

Accordingly, it win be appredated fiom Figures 3. 4 and 5 that the processing set may have man^ 
different forms and that the particular choice of a particular processing set st^ 

Oteprocessingrequirementofapardcularapphcationandthedegnreofredund^^ In the followmg 

description, it is assumed d»t the processing sets 14 and 16 refinied to have a strucm^ 

15 although it wUl be appreciated that anodicr form of processing set could be provided. 

niebridge(s) 12 are operable in a number of operating modes. These modes of operation will be 
described in more detail later. However, to assist in a general understanding of the structure of the bridge, the 
two operating modes will be briefly summarized here. In a first, combined mode, a bridge 12 is operable to 
n»te addresses and da^i be^veen the proces^ng «^ U and 16 (via the PA «»i PB b^ 

20 ,espectively)and.hedevice8(via.heDb«.22).In«hiscombinedmode.I/Ocyclesgen^ 

sea 14 and 16 are compared to ensure ftat both p,oces«ng sets are operating correctly. Comparison Mures 
force the bridge 12 into an error limiting mode (Estate) in which devfc^ 

information is collected. In the second, split mode, the bridge 12 routes and arbitrates addresses ««1 4^ 
one of the processmg sets 14 and 16 onto the D bus 22 and/or onto the other one of the pn^es^ 
25 14 respectively. In this mode of openuion. d« processing sets 14 arui 16 are not synctaonired and no 

co„,pari«„„ are made. DMA operations are also permitted m both modes. As mentioned above, the different 
„«Hles of operation, inch«iing the combined and spUt modes. wiU be desert However. 

there now follows a description of the basic structure of an example olibe bridge 12. 

Fig«„ 6 is a schematic fimctional overview of the bridge 12 of Figure 1. First and second processmg 

30 setI/ObusintertoPAbusin.erfi«.84andPBb«sin.erfi«86.arecom«^edtothePAandPBb^ 
and26 respectively. A device I«) bus interfece. D bus interfece 82. is competed to the D bus 22. Itshouldbe 
noted that the PA, PB and D bus mterfi«« need not be configured as separate elements but could be 
incorporated in other elements of the bridge. Accordingly, withm the context of .his docmnent. where a 
references is made to a bus interfece. tins does nm require d« presence of a specific sep^^ 

35 rato Ae capabihty of the bridge to connect m the bus concerned, for example by means of physical or logical 

bridge connections for die lines of the buses concerned. 

Routing (hereinafter termed a routing matrix) 80 is cmmected via a first internal path ^ 

interface 84 and via a second internal pa* 96 to ti«PB bus interfiice 86. THe routing matrix 80 is further 

connected viaadurd internal pad. 92 to d^Dbusmterface 82. The routingmatrix 80 is t^^^ 
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UO bus transaction routing in both directions between the PA and PB bus 

pro vide routing in both directions between one or both of the PA and PB bus intetfeces and the D bus m 
82. nie routing matrix 80 is connected via a further internal path 100 to storage control logic 90. THe storage 
control logic 90 controls access to bridge registers 110 and to a random access memory (SRAM) 126. Tbc 
5 routing matrix 80 is ti«refore also operable to provide routing in both directions between the PA. PB and D bus 
interfeces 84. 86 and 82 and the storage control logic 90. TUc routing matrix 80 is controUed by bridge control 
logic 88 over control padis 98 and 99. Hie bridge control logic 88 is responsive to control signals, data and 

addresses on internal paths 93. 95 and 97. and also to clock signals on the clock line(s) 21. 

In the embodiment of the invention, each of the P buses (PA bus 24 and PB bus 26) operates under a 

10 PCI protocol. The processing set bus controUers 50 (see Figure 3) also operate under the PQ protocol. 
Accordingly, the PA and PB bus inter&ces 84 and 86 each provide all the fimctionality required for a 
compatible interface providing bod. master and slave operation for data transferred to and fiom the D bus 22 or 
internal memories and registers of the bridge in the storage subsystem 90. Tie bus interfaces 84 and 86 can 
provide diagnostic infonnation to internal bridge status registers in the storage 

15 bridge to an error state (EStale) or on detection of an I/O error. 

TTxc device bus interfece 82 performs aU the functionality required for a VCl compliant master and 
sUve imerface for transferring daui to and fiom one of die PA and PB buses 84 and 86. ^ D bus 82 is 
operable during direct memory access (DMA) transfers to provide diagnostic inform^ 
registers in the Storage subsystem 90 of the bridge on transition to an Estate or on delert^ 

20 Fignre 7 ilh^rtrates in more detail the bridge legisten 110 and the SRAM 124. Tie storage control 

logic 110iscomiectedviaap«th(e.g. abas) 112 to a mm>ber of register components 114. 116, 118, 120. The 
storage control logic is also competed via a path (e.g. a bus) 128 to the SRAM 126 in which a posted write 
buffercomponentmandadirtyRAMcomponentmaremapped. Although a particular configuration of the 
components 1 14. 1 16. 118. 120. 122 and 124 is shown in Figure 7. these components may be configured m 

25 od« ways, with other components defined as regions of a common memory (e.g. a random access memory 

such as the SRAM 126. with the path 1 12/128 being formed by ti.e imemal addressing of the regK«s of 
memory). As shown in Figure 7. the posted write buffer 122 and the dirty RAM 124 are mapped to different 
regiom of the SRAM memory 126. whereas the registers 114. 116. 118and 120 are configured as «^ 

to SRAM memory. 

30 Control and stams registers (CSRs) 114 form inten-dregistera which allow d« 

operating modes of the bridge, allow the capture of diagnostic information for an Estate and for I/O 

control processing set access to pa slots and devices competed to die D bus 22. Tl«se regbters are set by 

signals from the routing matrix 80. 

Dissimilar data registera (DDRs) 116 provide locations for containing dissimilar data for different 
35 processing sets to enable nonnleterminisric data evems to be handled. TThssc registera are set by signals fiom 

the PA and PB buses. 

Bridge decode logic enables a common write to disable a da^ 

DDRs 11 6, one for each processing set 14 and 1 6. 
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A selected one of the DDKs can then be read in-sync by the processing sets 14 and 16. The DDKs thus 
provide a mechanism enabling a location to be reflected fiom one processing set (14/16) to another (16/14). 

Slot response registers (SRRs) 1 1 8 determine ownership of device slots on the D bus 22 and to aUow 
DMA to be routed to the appropriate processing set(s). These registers are linked to address decode logic. 

Disconnect registers 120 are used for the storage of data phases of an I/O cycle which is aborted whUe 
data is in the bridge on the way to another bus. TT« disconnea 120 receive aU data 

when a target device disconnects a transaction, or as the Estate is detected. These registers are connected to the 
routing matrix 80. Tl« routing matrix can queue up to three data words and byte enables. Pmvided the init«l 
addresses are voted as being equal address target controllers derive addresses which increment as data « 
exchangedbetweenthebridgeandthedestination(ortarget). Where a writer (for example a processor I/O write, or 
a DVMA (D bus to P bus access)) is writing data to a target, this data can be caught in the b^^ 
occurs. Accordingly, this data is stored in the disconnect registers 120 when an error occurs. TTtese disconnect 
registers can then be accessedon recovery ftom an Estate to recover the data as«^ 
which was in progress when the Estate was initiated. 

Although shown separately, the DDKs 116, the SRRs 118 and the discomiect registers may form an 

integiai part of die CSRs 1 14. 

Estate and error CSRs 1 14 provided for dte capmrc of a Ming cycle on the P buses 24 and 26. wifli an 

mdication of d« failing damm. Following a move to an EState, all of the writes initiated to the P buses are 
togged in die posted write bufer 122. Tlusse may be other writes durt have been posted in the processing set 
bus controllers 50. or which may be initiated by software beto an Estate internipt ^ 

stop carryii% out writes to the P buses 24 and 26. 

A diily RAM 124 is used to indicate which pages of d>e main memoiy 56 of die processing sets 14 and 

16 have been modifiedby direct memory access (DMA) transactions fiomone or more devices on 4^ 

Each page (e.g. each 8K page) is marlxd by a single bit in die dirty RAM 124 which is set when a DMA 

occurs and can be cleared by a read and dear cycle initiated on d« dirty RAM 124 by a 

processing set 14 and 16. 

The dirty RAM 124 and d« posted write buffer 1 18 may bofli be mapped into the naemory 124 in die 
bridge 12. Thismcmoryspacecanbeaccesseddnringnormalreadandwritecyclesfortestingpurposes. 
Figure 8 is a schematic fimctional overview of dte bridge control logic 88 shown in Figure ^ 
All of the devices comiected to AeD bus 22 are addressed geographically. Accordingly, dte bridge 
carries out decoding necessary to enrf,le the isohrting FETs to each slot befo^ 

initiated. 

Tlie address decoding performed by die address decode logic 136 and 138 essentially permits four 

basic access types: 

. an out-of-sync access (i.e. not in the combined mode) by one processing set (e.g. processing set 14 of 
Figure 1) to the other processing set (e.g. processing set 16 of Figure I), in which case the access is routed fiom 
thcPAbus interface 84 to AePB bus intcrfiM^e 86; 
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- an access by one of the processing sets 14 and 16 in the spUt 

in the combined mode to an I/O device on fl« D bus 22. in which case the access is routed via the D bus 

interface S2; 

- a DMA access by a device on the D bus 22 lo one or both of the processing sets 14 and 16. which 
would be directed to bod. processing sets 14 and 16 in ±e combined mode, or to the relevant processing set 14 
or 16 if out-of-sync and if in a spUt mode to a processing set 14 or 16 which owns a slot in which die device is 
located; and 

- a pa configuration access to devices m I/O slots. 

AS mentioned above, geographic addressing is employed. Hius. for example, slot 0 on motherboard A 
has the same address when referred to by processing set 14 or by processing set 16. 

Geographic addressing is used in combination with the PQ slot FET switching. During a 
configuration access mentioned above, separate device select signals are provided for devices which are not 
FET isoUteAAsingle device select signal can be provided for the switched pa slots as the FET 
used to enable a correct card. Separate FET switch lines are provided to each slot for separately switching die 

FETs for die slots. 

nui SRRs 1 18, which could be mcorporated in the CSR registers 114, are associated with die address 
decode fiinctions. TT^ SRRs 118 serve in a number of diferent roles which wiU be described in more detail 
later. However, some of the roles arc summarized here. 

In a combined mode, each slot may be disabled so that writes arc simply acknowledged without any 
transaction occnirring on the dcWce bus 22. whc«^ Reads will return meaningless data, once 

again witiiout causing a transaction CO Ac device board. 

In die spUt mode, each dot can be in one of three states. Hic statra are: 

-Not owned; 

-Owned by processing set A 14; 
* Owned by processing set B 16. 

A slot that is not owned by a processing set 1 4 or 1 6 making an access (this inch«ies not owned or un- 
owned slots) cannot be accessed. Accordingly, such an access is aborted. 

When a processing set 14 or 16 is powered ofi; an stots owned by it move to the unowned state. A 

processing set 14 or 16 can only claim an un^««ed stot. it camKrt wrest owM^ 

processingset TTris can only be done by powering off flie other processing set. or by getting die otfier 

processing set to reliiKiuish owneish^. 

The ownership bits are assessable and settable while in the combined mode, but have no efect until a 
split state is entered. This aUows die configuration of a spUt system to be determined while stiU in the 

combined mode. 

Each pa device is aUocated an area ofdie processing set address map. TTie top bits of die address are 
determined by d« Pa slot Where a device carries out DMA, the bridge is able to check that the device is 
using die correct address because a D bus arbiter informs the bridge which device is usmg die bus at a particular 
time If a device access is a processing set address which is not valid for it, dien die device access 
ignored It shouW be noted dat an address presented by a device wUl be a virtual address which wo^ 
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ttanslated by an I/O memory management unit in .he processing set bus conm,Uer 50 to an aemal memoi>- 
address. 

n»e addresses output by the address decoders ate passed via the initiator and target controllers 138 and 
140 to the routing matrix 80 via the hues 98 under control of a bridge controller 132 and an arbiter 134. 
5 An arbiter 134 is operable in various different modes to arbitrate for use of the bridge on a f»st-come- 

first-served basis using conventional PQ bus signals on the P and D buses. 

Inacombinedmodc, die arbiter 134 is operable to arbitrate between the in-sync processing sets Hand 
16 and any initiators on the device bus 22 for use of the bridge 12. Possible scenarios art: 

- processing set access to die device bus 22; 

jQ . processing set access to internal registers in the bridge 12; 

- Device access to die processing set memory 56. 

In spUt mode, both processing sets 14 and 16 must arbitrate the use of the bridge and thus access to the 
device bus 22 and internal bridge registers (e.g. CSR regis«rs 114). TTie bridge 12 must also contend with 
initiators on die device bus 22 for use of flat device bus 22. 
15 Each slot on the device bus has an arbitration enable bit associated with it. Tliese arbitration enable 

bits are cleared after reset and must be set to allow a slot to request a bus. When a device on the device bus 22 
is suspected of providing an I/O error. tf« arbitration enable bit for dat device is auto^ 

bridge. 

A pa bus intcrfece in dus processing set bus controlMs) 50 expects to be die master bus conlioto 
20 for d«sP bus concerned, that is it contains d«s pa bus arbiter the PA or PB bus to which tt is com«^ 
m bridge 12 camiot directly control access to the PA and PB buses 24 and 26. Tl« bridge 12 competes for 
access to diePAorPB bus with the processing set on fl« bus concenied under the comrol of d« bus 

50 on the bus concemed. 

Also shown in Figure 8 is a comparator 130 and a bridge controUer 1 32. The comparator 1 30 is 
25 opend,le to compare I/O cycles fiomd« processing sec. 14 and 16 to detennim^ any ontM^^ On 
determining an out-of-sync event, die compamtor 130 is operrfile to cause d« bridge controller 132 to acdvate 
an Estate for analysis of die out^f-sync event and possible recovery diereftom. 
Figure 9 is a schematic functional overview of die routing matrix 80. 
Tlie routing matrix 80 comprises a multiplexer 143 which is responsive to initial 
30 fron, die initiator controUer 138 of Figoie 8 to select one of die PA bus path 94. PB bus pad. 96. D bus pa* 

or internal buspadilOOasdiecurrentinputtod« routing matrix. Separate output buffers 144. 145. 146 and 
1 47 are provided for output to each of die pad« 94. 96. 92 and 1 00. widi diose buffte being sel^^^^ 
by signals 99 fiom die target controUer 140 of Figure 8. Between the multiplexer and the buffers 144-147 
signals are held in a buffer 149. in d« present embodiment d«e cycles of data for anI/0 cycle wUl be held m 

35 die pipeUnc represented by die multiplexer 143. die buffer 149 and die buffers 144. 

In Figures 6 to 9 a functional description of eleinents of die bridge has been given. Figure 10 is a 

schematic representation of a physical configuration of die bridge in which die bridge control logic 88. die 
storage control logic 90 and die bridge registers 110 are implemented in a first field programmable gate array 
(FPGA) 89. die routing matrix 80 is implemented in fimher FPGAs 80.1 and 80.2 and die SRAM 126 « 
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implemented as <«e or nu,« septate SRAMs addressed by a 

84 and 86 shown in Figi« 6 arc no. sepanite clenienu. but are int^ 

FPGAs 80.1 and 80.2 are used for the upper 32 bits 32-63 of a 64 bit PQ bus and the lower 32 bits 0-31 of the 
64bitPCIbus. It wiU be appnxiated that a singkFPGA could be employed for the routing nuttix 80 wh^ 
the necessary logic can be acconm^odated within the device. Indeed, where a FPGA of sufficient capacity is 
available, the bridge control logic, ston^e control logic and the bridge registers could be incorporated u. the 
same FPGA as the routing matrix. Indeedmanyotherconfigurationsmay be envisaged, and indeed technology 
other than FPGAs. for example one or more Applicarion Specific Integrated Circuits (ASICs) may be 
employed. As shown in Figure 10. the FPGAs 89. 80.1 and 80.2 and the SRAM 126 are competed via intemd 

bus paths 85 and paA control lines 87. 

Figure 11 is a transition diagram iUustniting in more detail the various operating m«^ 

The bridge operation «m be divided into three basic modes, namely an error state (EState) mode 150. a spUt 
statemode 156 and a combined state mode 158. The EState mode 150 can be further divided into 2 stat«.. 

After initial resetting on powering up the bridge, or foUowing an out^f sync event, the bridge is in t^ 
imtial Estate 152. In this state, all writes are stored in the posted write buffer 120 and reads from the internal 
bridge registers (e.g.. the CSR registers 116) are allowed and all other reads «e treated as enors (..e. they are 
aborted) In this state, the individual processing sets 14 and 16 perform evah««ions for dete™ 
time Eachprocessingsetl4andl6wiadetermineitsownres.arttimertiming.Thetimers^ 
"bbme-facorfbrthetnmsirionto die Estate. A processing set which determines that it is Ulcdy to have caused 
.heerrorsetsalongtimeforthetimer. A processing set which thinks it unlikely to have caused the em,r sets a 
shontin^fbrthetimer. The to pn^essing set 14 ««i 16 which times out. becomes a primary pn«^ 
Accordingly, when this is detenmned, the bridge moves (1 53) to the primary Estate 1 54. 

V^eitherprocessingsetl4/16hasbecometheprimaryprocessingset. the bridge is diea opeiatmg 
in Ae primary Estate 154. This state allows the primary processing set to write to bridge registets (specificaD 
theSRRsll8). OAerwri.es are no longer stored in the posted write buffer, but arc simply lost Devicebus 

reads are still aborted in the primary Estate 154. 

Once d«ESu«e condition is removed, .he bridge then moves (155) to die sptoswe 156. Inthespht 

state 156 accessu, ^device bus 22 is controUed by the SI» register. llSwhik access to^ 

simply arbitrated. The primary status of 4e processing sets 14 and 16 is ignored. Tranrition to a combmed 

operation isachievedbymeansofasync.resetdST). After issue of the sync.reset operation, the bridge uti«n 

,,pe„A,e in ti« combined «ate 158. whereby aU read «.! write accesses on the D bus 22 «^ 

buses 24 and 26 are allowed. All such accesses on the PA and PB buses 24 and 26 are . compared m the 

comparator 130. Detection of a mismatch ber»^ any read and write cycle, (with an exception of 

dissimilar data I/O cycles) cause a transition 151 to die EState 150. The various states described are controlled 

by Ac bridge controller 132. 

The role of the comparator 130 is to monitor ami compare I/O operations on die PA and PB buses m 

.he combined state 151 and. inresponsetoamismatchedsignaU to notify d« bridge contt^^ 

bridge comroller 132 causes die transition 152 to ti.e error state 150. The I/O operations can inch«le aU I/O 
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operations initiated by the processing sets, as well as DMA transfers in respect of DMA initiated by a device on 

the device bus. 

Table 1 below smmnarizes the various access operations which are aUowed in each of the operational 

states 

TABLE 1 





D Bus -Read 


D Bus- Write 


Estate 


Master Abort 


Stored in Post Write Buffer 


Primary Estate 


Master Abort 


Lost 


Split 


Controlled by SRR bits 


Controlled by SRR bits 


and arbitrated 


and arbitrated 


Combined 


Allowed and compared 


Allowed and compared 



As described above, after an initial reset, the system is in the initial EState 152. In dm state, neither 
processing sets 14 or 16 can access the D bus 22 or the P bus 26 or 24 of the odier processing set 16 or 14. Tbe 

imetnal bridge registers 1 16 of die bridge are accessible, but are read only. 

A system running in Ae combined niode 158 tnuisitions to die Estate 150 where diere i^ 

failure detected in this bridge, or altanatively a comparison Mure is detected in anodier bridge in a multi- 
bridge system as shown, for example, in Figure 2. Also, transitions to an EState 150 can occur in odier 
situations, for cxainple in the case of a sofhme controlled evem fiaming part of a 

On moving to die EState 150. an interrupt is signaled to aU or a subset of die processors of die 
processing sets via an interrupt line 95. Following diis. all I/O cycles generated on a F bus 24 or 26 result in 
reads being renirned widi an exception and writes being recorded in die posted write bu^^ 

Tbt operation of die comparator 130 will now be described in more detail. The comparator is 
comiected to pad« 94. 95. 96 and 97 for comparing address, daui and selected comrol signak from die PA and 
PB bus interfiiccs 84 and 86. A failed comparison of in-sync accesses to device I/O bus 22 devices causes a 

move from die combined state 158 to die EState 150. 

For processing set I/O read cycles, die address, command address parity, byte enables and parity ^ 

parameters are compared. 

If die comparison foils during die address phase, die bridge asserts a retry to die processing set bus 
controllers 50. which prevents data leaving die I/O bus comroUers 50. No activity occurs in diis case on die 
device I/O bus 22. On die processor(s) retrying, no error is returned. 

If die comparison fails during a data phase (only control signals and byte enables are checked), die 
bridge signals a target-abort to die processing set bus controUera 50. An error is returned to die processon. 

In die case of processing set I/O bus write cycles, die address, command, parity, byte enables and data 
parameters are compared. 
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If the comparison foils during the address phase, ihe bridge asserts a retry to the processing set bus 
controllers 50. which results in the processing set bus controUers 50 retrying the cycle again. The posted write 
buffer 122 is flicn active. No activity occurs on the device UO bus 22. 

If die comparison fails during die data phase of a write operation, no data is passed to the D bus 22. 
The failing data and any other transfer attributes fiom both processing sets 14 and 16 arc stored in the 
disconnect registers 122. and any subsequent posted write cycles are recorded in the posted write buBer 118. 

In die case of direct virtual memory access (DVMA) reads, die data control and parity are checked for 
each datum. If die data does not match, the bridge 12 terminates die transfer on die P bus. In die case of 
DVMA writes, control and parity error signals are checked for correctness. 

Odicr signals in addition to diose specifically mentioned above can be compared to give an indication 
of divergence of die processing sets. Examples of diese are bus grants and various specific signals during 
processing set transfers and during DMA transfers. 

Errors M roughly into two types, diose which are made visible to die software by die processing set 
bus controUer 50 and diose which are not made visible by die processing set bus controUer 50 and hence need to 
be made visible by an interrupt ftom die bridge 12. Accordingly, die bridge is operable to capture errors 
reported in connection widi processing set read and write cycles, and DMA reads and writes. 

Qock control for die bridge is performed by die bridge controller 132 in response to die clock signals 
&om dK clock line 21. Individual control lines ftom dns controller 132 to die various elements of die bridge are 

not shown in Kgures 6 to 10. 

Figure 12 is a flow diagram ilhistiating a possible sequence of operating stages where lock«^ 

are detected during a combined mode of iteration. 

Stage SI represents die combined mode of qpciation where lockstqi error checking is performed by 

the comparator 130 shown in Figure 8. 

In Stage S2, a lockstep error is assumed to have been detected by die comparator 130. 

In Stage S3, die curnmt state is saved in die CSR registers 114 and posted writes are saved in t^ 

posted write buffer 1 22 and/or in die disconnect registers 1 20. 

Figure 13 iUnstrates Stage S3 in more detaiL Accordingly, in Stage S31. die bridge controUer 132 
detects whedicr die lockstep error notified by die comparator 1 30 has occurred during a data phase in which it is 
possible to pass data to die device bus 22. In tins case, in Stage S32, die bus cycle is terminated. Then.inSlage 
S33 die data phases are Stored in die disc<mne« registers 120 and control dien passes to St«e 
evahiation is made as to whedw a fhrdier I/O cycle needs to be stored. Ahematively, if at Stage S3l. it is 
determined tiiat die lockstep error did not occur during a data phase, die address and data pha«^ 
write I/O cycles are Stored in die posted write buffer 1 22. At Stage S34. if dieie are any further posted write I/O 
operations pending, tiiese are also stored in die posted write buffer 122. 

Stage S3 is performed at die initiation of die initial error state 152 shown in Figure 1 1. In diis state, die 
first and second processing sets arbitrate for access to die bridge. Accordingly, in Stage S31-S35. die posted 
write address and data phases for each of die processing sets 14 and 16 are Stored in separate porA^ 
posted write buffer 122, and/or in die single set of disconnect registers as described above. 
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Figure 14 illustrates the source of the posted write I/O cycles which need to be stored in the posted 
write buffer 122. During norn«d operation of the processing sets 14 and 16. output buffers 162 in the individual 
processors contain I/O cycles which have been posted for transfer via the processing set bus controUers 50 to 
the bridge 12 and evenn«Uy to the device bus 22. Sinritoriy. buffers 160 in the processing set controUers 50 
also containposted I/O cycles for transfer over the buses 24 and 26 to the bridge I2andevenluaUy to th^ 

bus 22. 

Accordingly, it can be seen dat when an error state occurs, I/O write cycles niay already have been 

posted by the processors 52, either in their own buffers 1 62, or aheady transferred to the buffers 160 of the 

processing set bus controUers 50. It is the I/O write cycles in the buffers 162 and 160 which graduaUy 

propagate Arough and need to be stored in die posted write buffer 122. 

As shown in Figure 15, a write cycle 164 posted to the posted write buffer 122 can comprise an 

address field 165 including an address and an address type, and between one and 16 data fields 166 inctading a 

byte enable field and die data itself. 

data is written into die posted write buffer 122 in die EState unless die initiating processing set has 
been designated as a prin««y CPU set. At that tirne. non-prinmy writes in an Estate stiU go to die posted 

buffer even after one of die CPU sets has become a primary processing set An address pointer in die CSR 
registers 1 14 points to dte next avaibble posted write buffer address, and also provides «i overflow bit which is 
set when dte bridge atteinpts to write past of dte top of the posted write buffer for any one of d* processing 

14 and 16. Indeed, m dte iHesent implemenlation. only the first 16 K of data is leco^ 
Attempts m write beyond flte top of the posted write buffer are ignored The vahte of dte ^ 
pointer can be cleared at reset, or by softwareusingawrittmider die control ofaim^ 

Returning to Figure 12. after saving tlte status and posted writes, at Stage S4 the faK^^ 
sets independently seek to evahBite d« error state and to determine whether one of the pro«^ 
Tins detemiination is made by dte individual piocessoni in an error state in wWch dx^ 
fiomdtecontrolstattandEStateregisterslM. During dris error mode, dte arbiter 134 arbitrates fcr access te 

±e bridge 12. 

hi stage S5, one of die processing sets 14 and 16 establishes itseff as die priniary processings This 
is determined by each of the processing sets identifying a time factor based on die estimated degree of 
„sp«udbaity for d« error, whaebydte first i«ocessing set to time out be«^ ^ 
Stage S5.dtestatns is recovered for fliat processing set and is copied to dteodier processings^ Theprimary 
processing is able to access the posted write buffer 122 and die disconnect registers 120. 

toStageS6.dtebridgeisoperableinaspUtmode. ff it is possible to rt^establish an equivalent status 
for dte first and second processing sets, dien a reset is issued at Stage S7 to put dte proc^ 
combmed mode at Stage SI. However, it may not be possible to r,>«tabUsh an equivalent state until a faulty 
processing set is rephiced. Accordingly dte system wiU stay in die Split mode of Stage 86 in order to continued 
operation based on a singte processing set After rephicmg die fimlty pmcessing set dte system could dien 

establish an equivalent state and move via Stage S7 to Stage SI. 

AS described above, die comparator 130 is operable in die combined mode to compare die I/O 
operations output by die first and second processmg sets 14 and 16. This is fine as long as aU of dte I/O 
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operations of the first and second processing sets 14 and 16 are fiiUy synchronized and deterministic. Any 
deviation fiom this will be interpreted by the comparator 130 as a loss of lockstep. TTus is in principle correct 
as even a minor deviation &om identical outputs, if not trapped by the comparator 130. could lead to the 
processing sets diverging further fiom each other as the individual processing sets act on ih^ deviating outputs. 
However, a strict application of this puts significant constraints on the design of the individual processing sets, 
.^n example of dus is that it would not be possible to have independent time of day clocks in the individual 
processing sets operating under their own clocks. Utis is because it is impossibk to obtain two crystals which 
are 100% identical in operation. Even smaU differences in the phase of die clocks could be critical as to 
whether the same sample is taken at any one time, for example either side of a clock transition for the respective 
processing sets. 

Accordingly, a solution to this problem employs ti« dissimilar data registers (DDR) 116 mentioned 
earhenTTie solution is to write data ftom the processing sets into nsspectiveDDIUi in 4^ 
rhe comparison of fl« data phases of the write operations and then to read a selected one of the DDRs back to 
each processing set, whereby each of die processing sets is able to act on the same data. 

Figure 17 is a schematic representation of details of die bridge of Figures 6 to 10. It will be noted diat 
details of the bridge not shown in Figure 6 to 8 are shown in Figure 17. whereas otinrr details of die bridge 
shown in Figures 6 to 8 are not shown in Figure 17, for reasons of darity. 

The DDRs 1 1 6 ans provided in die bridge registers 1 10 of Figure 7. but could be provided elsewhere in 
d« bridge in odier embodiments. One DDR 116 is provided for each processing set In die example of die 
multi-processor system of Figure 1 where two processing sets 14 and 1 6 are provided, two DDRs 1 1 6A and 
1 1 6B are provided, one for each of die first and second processing sets 14 and 16. respectively. 

Figure 17 represents a dissimitor data write stage. Tte addressing logic 136 is shown schematically to 
comprise two decoder sections, one decoder section 136A for d« first processing set and «^ 
136Bfordiesecondprocesshigsetl6. Duringanaddressphaseof a dissimihir data I/O write operation each of 
du: processing sets 14 and 16 outputs die same predetermined address DDR-W which is sepam 
by die respective first and second decoding sections 136A and 136B as addressing die r«^ 
secondrespectiveDDRsll6Aandll6B. As die same address is output by die first and second processing sets 
14 and 16. tius is not interpreted by die comparator 130 as a lockstep error. 

TT« decoding section 136A. or d« decoding secticm 136B. or bodi are arranged to fiudier oui^ 

disabk signal 137 in response to die predetermined write address supplied by d« firs^ 

set8l4andl6. TTiis disable signal is supplied to d« comparator 130 and is operative dnri^ 

die write opemtion to disable die comparator. As a result, die data output by die first processing set can be 

stored in d« first DDR 116A and die data output by die second ptoccssing set can be stored in dK 

1 16B widiout die comparator being operative to detect a difference, even if die data ftom die first and second 

processing sets is different m first decoding section is operable to cause d« routing matrix to store die data 

ftom die first processing set 1 4 in die first DDR 1 1 6A and die second decoding section is operable to cause die 

muting matrix to store die data ftom die secoml pnx«ssing set 16 in die second DDR 1 1 63. At die end of die 

data phase die comparator 130 is once again enabled to detect any differences between I/O addr^ 

phases as indicative of a lockstep error. 
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FoUowing the writing of the dissinrilar data to the fnst and second DDRs 116A and 116B. die 
processingsctsaredienopend.letB«addiedataftomaselectedoneofdieDDRs 116A/n6B. 

Figure 18 illustrates an alternative arrangement where die disable signal 137 is negated and is used to 
control a gate 131 at d« output of die comparator 130. When die disable signal is active die output of dw 
5 comparator is disabled, whereas when d« disable signal is inactive die output of die comparator is enabled. 

Figure 19 ilhislrates die reading of die fint DDR 1 16A in a subsequent dissimilar data read stage. As 
illustrated in Figure 19. each of die processing sets 14 and 1 6 outputs dtt same predetermined address DDR-RA 
which is separately interpreted by die respective first and second decoding sections 136A and 136B as 
addressing die same DDR, namely die f«st DDR 1 1 6A. As a result, die content of die first DDR 1 16A .s read 
10 by bodi ofdie processing sees 14 and 16. diereby enabling diose processing sets to receive d«s^ Th»s 
««bles die two pmcessing sets 14 and 16 to achieve detemrinistic behavior, even if die souree of d« daa 
wrinen into die DDRs 1 16 by die processing sets 14 and 16 was not deterministic. 

AS an alternative, die processing sets could each read die data ftom die second DDR 116B. Figure 20 
illustrates die reading of die second DDR 116B in a dissimilar data read stage foUowing die dissimilar data 
15 write stage of Figure 15. As ilhistrated in Figure 20, each of die processing sets 14 and 16 outputs die same 

predetermined address DDR-RB which is separately interpreted by die respective first and second decodmg 
sections 136A and 136B as addressing die same DDR, n«ncly die second DDR 116B. As a result, die content 

of die second DDR 1 16B is read by bodi of die processing sets 14 and 16. d«eby enabling d«se precessmg 
sets .0 receive die same data. As widi die dissimilar data read sn«e of Figmt 16, dus enables die two 
20 pnKXSsing»elsl4andl6to«duevedeternnnisticbehavior.evenifdie»^ 

1 16 by die processing sets 14 and 16 was not detemnnistic. 

TTie selection of which of die first and second DDRs 1 1 6A and 116B to be read can be determined in 

anyappropriaren^mnerbyd^soflwareoperatingondieprocessingmodules. Hds could be done on d« basis 
of a simple selection of one or d« od« DDRs. or on a statistical basis or randomly or in any odier mamiw 

25 long as die same choice ofDDR is made by bodi or all ofdic processing sets. 

Figure 21 is a flow diagram summarizing die various stages of operation of die DDR mechanism 

described above. 

In stage SIO. a DDR write address DDR-W is received and decoded by die address decoders secnons 

136A and 136B during die address phase of die DDR write operation. 

30 In stage SI 1, die comparator 130 is disabled 

In stage 812. die data received ftom dK processing sets 14 and 16 during die data phase of d« DDR 

write operation is stored in d»s first and secondDDRsll6Aandll6B. respectively, as selected by 

second decode sections 136A and 136B, respectively. 

m stage S13. a DDR read address is received ftom die first and second processing sets and is decoded 
35 by die decode sections 136A and 136B. respectively. 

If die received address DDR-RA is for die first DDR 1 16A. dien in stage S14 die content of diat DDR 
116A is read by bodi of die processing sets 14andl6. 

Ahematively. 116A if d« received address DDR-RB is for die second DDR 116B. dien in stage S15 
die content of diat DDR 1 16B is read by bodi of die processing sets 14 and 16. 
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Figure 22 is a schematic icpresentation of the arbitration performed on the respective buses 22. 24 and 

26, and the arbitration for the bridge itself. 

Each of the processing set bus contn,Ucrs 50 in the respective processing sets 14 and 16 inchidcs a 
conventional PO master bus arbiter 180 for providing arbitration to the respective buses 24 and 26. Each of the 
Hester arbiters 180 is responsive to request signals ftom .he associated processing set bus controUer 50 and the 
bridge 12 on n«pective request (REQ) lines 181 and 182. The master arbiters 180 aUocate access to the bus on 
a fust^ome-fust-served basis, issuing a giant (CSNT) signal to the winning party on an appropriate grants hne 
183 or 184. 

A conventional VCl bus arbiter 185 provides arbitration on the D bus 22. The D bus arbiter 1 85 can be 
configuredas part oftheDbus interface 82 ofFigure6orcould be separate therefiom. As with the P bus 
master arbiters 180. the D bus arbiter is responsive to n^quest signah ftom the c^ 

bridge and the devices 30, 31. etc. connected to the device bus 22. Respective request lines 186, 187, 188. etc. 
for each of the entities competing for access to the D bus 22 are provided for die request signals (REQ^^ 

bus arbiter 185 aUocates access to the D bus on a fiist^ome-first-served basis. ^ 

the winning entity via re^ective grant lines 189. 190. 192. etc. 

Figure 23 is a state diagram summarising the operation OftheDbus arbiter 185. In a particular 

cmbodimentup to six request signals may be produced by respectiveDbus devices a^ 

On a transition into die GRAOT state. tf«sc are sorted by a priority encoder and a requ«^ 

Ae highest priority is registered as tf« winner and gets a grant (GNT«0»«^ Each wim« which is selected 

„„Hlifi«.heprioritiesiaapriori.yencodersoth«givend»same^^ A 

diffaentdevicehastbeW^prio.ity.hH^eachdevKehasa'^d^ T^^^ 
REQ# has a higher weighting dum D bus deWces and will, mute very busy condMons. get the ^ 

second device. 

If a device requesting the bus finis to perfbon a transaction widtin 1 6 cycles it may lose 
BACKOFF state. BACKOFF is required as. mutaParnles. a device may access the bus one cycfcafierGm-# 
isremoved. Devices may only be granted access to D bus if the bridge is not in the not in the EState. Anew 
GNT# is produced at Ae times when die bus is idle. 

In the GRANT and BUSY states. ti»e FEIs are enabled and an accessing device is known and 
forwarded to tfKDbus address decode togic for checking againstaDMAaddnsspr^ 

Turning m.w to the bridge arbiter 134. this allows access to the bridge for 
ti.ePaFRAME#sigmdindicatinganadd«sphase. Figme 24 is a state diagram smmnarising die operation of 

die bridge arbiter 134. 

As widi the D bus arbiter, a priority encoder can be provided to resoWe access attempts wWch w^^^ 

In diis case "a collision" the loseirtoseis are retried which forces them to give up die bus. Under PQ rules 
retried devices must try repeatedly to access die bridge and diis can be expected to happen. 

To prevent devices which are very quick wifli ti«ir retry attempt ftom hogging die bridge. retru«i 
interfaces are remembered and assigned a higher priority. These remembered retr^^ 

wayasaddressphases. However as a precaution tins mechanism is timed out so as mit to get smckwaitmg for a 
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faulty or dead device. TTie algorithm employed prevents a device which hasn't yet been retried, but which 
would be a higher priority retry than a device cunently waiting for. fiom being retried at the first attempt 

to combined operations a PA or PB bus input selects which P bus intcrfiice wUl win a bridge access. 
Both are informed they won. Allowed selection enabks latent fault checking during normal operation. EState 

prevents the D bus from winning. 

nie bridge arbiter 134 is responsive to standard PCI signals provided on standard PCI control lines 22. 

24 and 25 to control access to the bridge 12. 

Figure 25 iUnstrates signals associated with an I/O operation cycle on the PCI bus. A PQ fiame signal 
(FRAME#) is initially asserted. At the same time, address (A) signals will be available on the DATA BUS and 
the appropriate command (write/read) signals (Q will be available on the command bus (CMD BUS). Shortly 
aftertheftamesignalbeingassertedlow.theinitiatorreadysignalCIRDY*)^ Whenthe 
device responds, a device selected signal (DEVSEL#) wUl be asserted low. When a target ready signal « 
asserted low (TRDY#). data transfer (D) can occur on Ae data bus. 

The bridge is operable to allocate access to the bridge resources and thereby to negotiate aUocation of a 
mrget bus in response to the FRAMM being asserted low for the initiator bus concern Accordingly, the 
bridge arbiter 1 34 is operable to allocate access to Am bridge resources and/or to a target bus on a fSrst-come- 
firet-served basis in response to the FRAMED being asserted low. As weU as the simple first-come-first-served 
basis, the arbiters may be additionally provided with a mechanism for logging to arbitt^ 
in^iy a conflict tesoh-ion based on the request and allocation 
idemicaltime. Alternatively, a simple priority can be alkn^ted to the various r« 
of identically timed requests,aparticularitqaestet always wins the aDocato 

Each of the slots on the device bus 22 has a slot response register (SRR) 11 8. as weU as other devK^ 

comiected to the bus. such as a SCSI interfu^. Each of the SRRs US contains bits defining 
d« slots, or the devices competed to du: slots on the direct memory access bus. In this embodiment, and for 
reasons to be elaborated below, each SRR 118 comprises a four bit register. However, it wiU be appreciated 
tfut a larger register wiU be required to determine ownership between more than two ptoc^ For 
example, if three processing sets are provided, then, five bit register will b,^ required for each slot 

Figure 16 iltosttates schematically one such four hit register 600. As shown in Figure 16. a first b.t 
602 is identified as SRRCOlasecond hit 604 is identified as SKRlllathird hit 606 is 

30 a fourth bit 608 is identified as SRR[3]. 

Bit SRRIO] is a hit which is set when writes for valid transactions are to be suppressed. 

Bit SRRIl] is set when the device slot is owned by the first processing set 14. This defines the access 
routebetweenthefirstprocessingset 14andthedeviceslot \»^thisbitisset.thefi^ 
always be master of a device slot 22. while the abiKty for the device slot to be master depends on whether b.t 

35 SRR[31isset ^. ^ ^ ^. 

Bit SRR[2] is set when the device slot is owned by the second processing set 16. This defines the 
access route between the second pmcessing set 16 «ul the device slot When this bit is set. the second 
processing set 16 can ahvays be master of the device slot or bus 22. whUe the ability for the device slot to be 
master depends on whedier bit SRR[3] is set 
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Bit SRRP] is an aAitralion bit which gives the device slot the ability to become nmster of the device 
bus 22 . but only if it is owned by one of the processing sets 14 and 1 6. that is if one of the SRR m 



bits is set 



When d« fa« bit (SRR[0]) of an SRR 118 is set. writes to the device for that slot a« ig^^^ 
no, appear on the device bus 22. Re«ls rennn indeteinrinate data without causing a tiansacdon on ±e 
bus 22 In die event of an I/O eitor the fake bit SRR(0] of the SRR 1 88 corresponding to die dev.ce which 
caused die em,r is set by die hardware configuration of die bridge to disable fimher access to die device slot 
concerned. An internriitn^yalsobe generated by diebridge to inform die software 

leading to die I/O error duit die error has occurred. THe fake bit has an effect whedier die system is in die spht 

or the combined mode of operatioiL 

nicownershipbitsonlyhaveeffect. however. indiespUtsystemmodeof operation. In diis mode. 

each slot can be in three states: 
Not-owned; 

Owned by processing set 14; and 
Owned by processing set 16 

This is determined by d« two SRR bits SRRIH and SRR12]. wid. SRRll] being set when die slot « 
ov«iedbyp.ocessingsetl4andSRRt2]beingsetwhendieslotisownedbyprocessingsetB^ If die slot is un- 
owned. d«n ncid« bit is set (bofli Wis set is an illegal condition and is prevented by d« h«^ 

A slot which is not owned by d« processing set making die access (dris inchides nnK^ 
cannot be accessed and results in an abort A processmg set can only daim an un-«wned slot; it cannot wrest 
ownershipawayftomanod«rproce«dngset This can only be done by poweri-g-offd^odierprocessmg set. 
When a processing set is powered off; an slots owned by it move to d«u.«^ Whilst it is not 

possible forapnH:essing set to wrestownershipfiomanod^proces^ 

give owncrshq) to ano&cr jwoccssing set 

Tbe owned bits can be altered when in die combined mode of operation state but duqr have ^ 

until the split mode is entered. 

Table 2below summarizes d« access rights as determined by an SRR118. 

Fmm Table 2. it can be seen diat when d« 4-bit SRR for a given device is set to 1 100. for example, 
tiien die stot is owned W processmg set B (ie. SRRC21 is logic high) and processor 

wricetod«device(Le.SRRlllisk.giclow).aldmghitmayreadftomorwritetoti«bri^^^ TAKE.AT« 
set logic low (i.e. SRRtO] is logic low) indicating duu access to die devi« 

on die bus. M "ARB EN" is set logic high (i.e. SRRPl is logic high), die device widi which die reg«« 
associated can become master of die D bus. TOs example demonstrates d« operation of die register when die 
bus and associated devices are operating correctiy. 
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SRR 
[3[2][1][0] 

0000 
xOOx 

0010 



0100 



1010 



1100 



0011 



0101 



1011 



UOl 



PA BUS 



TABLE 2 
PB BUS 



Device Interface 



ReadWrite bridge SRR Read/Write bridge SRR Access denied 



Read/Write bridge 
Owned D Slot 

Read/Write bridge 
No access to D Slot 

Read/Write bridge. 
Owned D Slot 

Read/Write bridge. 
No access to D Slot 

Read/Write bridge. 
Bridge discard writes 

Read/Write bridge. 
No access to D sbt 

Read/Write bric^ 
Bridge discard writes 

Read/Write bridge. 
No access to D slot 



Read/Write bridge 
No access to D Slot 

Read/write bridge 
Access to D Slot 

Read/Write Bridge 
No access to D Slot 

Read/Write bridge 
Access to D Slot 

Read/Write bridge 
No access to D Slot 

Read/Write bridge 
Bridge discards writes 

Read/Write bridge 
No access to D Slot 

Read/Write bridge 
Bridge discards writes 



Access Denied because 
arbitration bit is off 

Access Denied because 
arbitration bit is off 

Access to CPU B Denied 
Access to CPU A OK 

Access to CPU A Denied 
Access to CPU B OK 

Access Denied because 
Arbitration bit is off 

Access Denied because 
Arbitration bit is off 

Access to CPU B Denied 
Access to CPU A OK 

Access to CPU B Denied 
Access to CPU A OK 



In an alternative cxarnple. where the SRR for the device is set to 01^^^ 
indicates that the device is owned by im)cessing set B. Ho^^ 

logic lowandthedcviccisnotaUowedaccesstotheim>cessi^ SRR[0] is set high so that any writes to die 
device are ignon^ and reads therefom return indeterminate^ In this way, the malfunctioning device is 
effectively isolated &om the processing set, and provides indeterminate data to satisfy any device drivers, for 
example, that might be looking for a re^xmse from the device. 
- Figure 26 ilhistrates the operation of the bridge 12 for directing 

the devices 28. 29, 30, 31 and 32 to the memory 56 ofthet>rocessing sets W When the D bus arbiter 

1 85 receives a direct memory access (DMA) request 193 ftom a device (e.g., device 30 in slot 33) on the device 
bus^theDbusarbiterdetermineswhethertoallocatethebustod^ As a resuU of this granting procedure, 
the D.bus arbiter knows the slot which has made the DMA request 1 93. The DMA request is supplied to the 
address decoder 142 in the bridge, where the addresses associate Tlie address 

decoder is responsive to the D bus grant signal 194 for the slot 

granted access to die D bus for die DMA request 

TT« address decode logic 142 holds or has access to a geographic addr^ 196. which identifies 

the relationship between the processor address space and die slots as a result of the geographic address 
employed. This geographic address map 196 could be held as a table in die bridge memory 126, along widi the 
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posted write buffer 122 and the dirty RAM 124. Alternatively, it could be held as a table in a separate memory 
element, possibly forming part of the address decoder 142 itself. Hie map 182 could be configured in a form 
Other than a table. 

The address decode logic 142 is configured to verify the correctness of the DMA address 
the device 30. In one embodiment of the invention, this is achieved by comparing four significant address bits 
of the address supplied by the device 30 with the corresponding four «ldress bits of the address held in the 
geographic addressing map 196 for the slot identified by the D bus grant signal for the DMA request In this 
example, four address bits are sufficient to determine whether the address suppUed is within the correct address 
range, to this specific example. 32 bit Pa bus addresses are used, with bits 3 1 and 30 always being set to 1. bit 
29 being aUocated to identify which of two bridges on a motherboard is being addressed (see Figure 2) and bits 
28 to 26 idemifying a Pa device. Bits 25^ define an offiet from the base address for the address range for 
each slot Accordingly, by comparing bits 29-26. it is possible to identify whether the addre8s(cs) supplied 
falKs) within the appropriate address range for the slot concenu«L It wiU be appreciated that in other 
embodiments a different number of bits may need to be compared to make 4is determination depem^ 

i the allocation of the addresses. 

The address decode logic 142 could be arranged to use the bus grant signal 184 for the slot concerned 
,0 identify a table entry forthc slot concerned and flicn to compare the address in that en^ 
received with the DMA request as described above. Alternatively, the «Uress decode togic 142 could be 
ananged to use the addnas(es) received with the DMA address to address a relational g^ 

, andlodetetmineaslotm«totheref««..whk*couWbecomp.redtothesto 

194 fa intended and then*y to determine whether the addresses M within the address ra^ 

slot concerned. 

mther way. the address decode logic 142 is arranged to perimt DMA to proceed if the DMA addressee 
fidl within the expected address space for the slot concerned. (M«^ 
25 ignore the slots and ti«i*ysical addresses. 

■nc address decode logic 142 is finther operable to control the routing of the DMA request to the 
appropriate processing set(s) 14/16. If the bridge ism the combined mode, the DMA access will automatically 
be allocated to aU of the m-sync processing sets 14/16. The address decode logic 142 will be aware that the 
bridge is in the combmed mode as it is under the control of the bridge controller 132 (see Figure i). However. 
30 whered.ebridgeisinthesplrtmode,adecisionwinneedtobemadeastowhich.ifany.offl^ 

the DMA request is to be sent 

When the system is m split mode, the access WiU be directed to a processing set 14 or 16 which owM 

the slot concerned. If the slot is un-owned. then the bridge does not respond to the DMA request Inthesplit 
mode, the address decode logic 142 is operable to determine the ownership of tht device orig^ 
35 request by accessing die SRR 118 for die slot concerned. The appropriate slot can be identified by the D bus 
g««rignaL Ti.eaddressdecodelogicl42isoperabletoconlrolthetargetcontn,llerl40(seeFig«^ 
theDMAnKpiesttotheappropriateptocessingset(s)14/16basedonlheownersto^ If 
bit SRR[1] is set. the first processing set 14 is the owner and the DMA request is passed to the fir« 
set If bit SRRPl is set. the secondprocessing set 16 is the owner and the DMA request is pass^^ 
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processing set If ncilbcr of the bit SRRtU and SRRt2] is set, then the DMA request is ignored by the address 
decoder and is not passed to ei&er of the processing sets 14 and 16. 

Figure 27 is a flow diagram sammarizing die DMA verification process as illustrated with reference to 

Figure 24. 

In stage S20, the D-bus arbiter 160 arbitrates for access to die D bus 22. 

In stage S21, the address decoder 142 verifies tbe DMA addresses supplied with the DMA request by 

accessing the geogrt^ihic address map. 

In stage S22, the address decoder ignores die DMA access where the address fi^ 

range for the slot concerned. 

Altcmativeiy. as represented by stage S23, the actions of the address decoder are dependent upon 

whedier the bridge is in die combhied or Ae split mode. 

If the bridge is in die combined mode, then in stage S24 the address decoder controls die target 
controller 140 (see Figure 8) to cause the routing matrix 80 (sec Figure 6) to pass the DMA re«piest to both 

processing sets 14 and 16. 

If the bridge is in die split mode, die address decoder is operative to verify die ownership of die slot 

concerned by reference to die SRR 118 for diat slot in stage S25. 

If slot is allocated to d« first processing set 14 (Le. the SRR[1] bit is set), then in stage S26 Ae 
address decoder 142 comreb d« target controller 140 (see Figure 8) to cause dus routing ma^ 

6) to pass the DMA request to first processing set 14. 

If the slot is allocated to die second processing set 1 6 (Le. die SRKP] bit is set), dicn in sB^ 

address decoder 142 controls d» target conUoUer 140(sec Figure 8) to cause d« routing matrix 80 (see Rgnre 
6) to pass die DMA request to die second processing set 16. 

If the slot is unallocated (Le. neidier d« SRIUll bit nor flic SRRP] bit is set). dKu m 

address decoder 142 ignores or disc«ds die DMA request and d« DMA request is not passed to fl«pre^ 

sets 14 and 16. 

A DMA. or direct vector memory access (DVMA). request sent to one or more of die processing sets 
causes die necessary memory operations (read or write as appropriate) to be effected on die p^ 

memory. 

There now follows a description of an example of a mechanism for enabling automatic recovery ftom 

an Estate (see Figure 11). 

m automatic recovery process mchides reintegration of die state of die precesshig sets to 

stamsinordertorttanptarestartinlockstep. To «diievedns.d« processing set which asserts itself as d« primary 
p™«singsetasdescribeddM«recopiesitscompletestatetod«odierp™ TOs involves ensuring dat 

die content of die memory of bodi processors is die same before trying a restart mbckstep mode. 

However, a problem widi die copying of die content of die memory fiom one processing set to die odier is 

diat during dus copying precessadevicecomHXtedtod«Dbus 22 might atlempttomalre 

(DMA)requestforaccesstodiememoryofd«p.ima,yprecessingset If DMA is enabled. d«i a write made to 

an area of memory whichhasabeady been copied wouldresuh in die memory state of d^twoproccK^ 

of die copy not being die same. In principle, it would be possible to inhibit DMA for die whole of die copy 
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process. Howem. dus would be undesiiable. bearing in 0ii«^ 

system or the lesomces of the system are unavailable. As an alternative, it would be possible to retiy the whole 
copy operation when a DMA opcmtion has occuned during die period of the copy. However, it is likely that 
junher DMA operations would be perfomjed during the copy letiy, and acc^ 

cither. Accordingly, in the present system, a dirty RAM 124 is provided in the bridge. As described earlier the 

dirty RAM 124 is configured as part of tiw bridge SRAM memory 126. 

TTie dirty RAM 124 comprises a bit map having a dirty indicator, for ejumiple a dirty bit, for each block, 
or page, of memory. The bit for a page of memory is set when a write access to the area of memory concerned is 
aade. to an embodiment of the invemion one bit is provided for every 8K page of main processing set m^ 
Tl« bit for a page of processing set memory is set automatically by the address decoder 142 ^ 
DMA request for that page of memory for either of the processing sets 14 or 16 ftom a device comiected to 
bus22. TTk dirty RAM can be reset, or deared when it is read by a processing set. for exampte by means of read 
and clear instructions at die begimiing of a copy pass, so tfiat it can start to record pages wW^ 

given time. 

The dirty RAM 124 can be read word by word. If a large word size is chosen for reading die dirty RAM 

124, diis will optimise die readmg and resetting of die dirty RAM 124. 

Accordingly, at du5 end of d» copy pass die bits in ti« dirty RAM 124 will indicate diose pages of 

praising set memory which have been changed (or dirtied) by DMA writes during dtep^ A 
(urdiercopypasscandieabeperitatedforonlyd««pagesofmemo.ywhichhavebee^ TOswilltake 
less time data fall copy of die memory. Accordingly, dieie are typically less pages marked as dirty at die end of 
diene«copypassand,asaresult,d«copypas8e»canbecome8horterandshorta. As some time it is necessary to 
decide to inhibtt DMA writes for a short period for a final short, copy pass, at die end of which dte memories o^ 
die two processing sets win be d« same and the primary processing set can issue a reset operation to re^ 

combined mode. 

TT« dirty RAM 124 is set and cleared in bodi die combined and spBt modes. This means diat in split 

mode the dirty RAM 124 may be cleared by citiier processii« set 

Tte dirty RAM 124 address is decoded ftom bits 13 to 28 of die pa address presented by die D bus 

device. Erroneousaccesseswbichp.esemillegalcombinationsofdteaddressbits29to31arem.^ 
dirty RAM 124 and a bit is dirtied on a write, even diough die bridge win not pass diesc tn^^ 

processing sets. 

When TOiding the dirty RAM 124, the bridge dc 

as diny lOVM and win clear the cjmtents of any locatim 

As an alternative to providing a single dirty RAM 124 which is cleared on being read, anodicr 

altcmaiivc would be to provide two dirty RAMs which are used in a toggle mode, with one being written to 

while anodier is read. 

Figure 28 is a ftow diagram surmnarising the operation of the dirty RAM 124. 

hi stage S41, the primary processing set reads the dirty RAM 124 w^ 
dirty RAM 124. 
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In stage S42, the primary processor (e.g. processing set 14) copies the whole of its memory 56 to the 

memoiy 56 of flie odier processing set (eg. processing set 16). 

In stage S43. the primary processing set reads the dirty RAM 124 which has the efie^ 

dirty RAM 124. 

to stage S44, the primary processor determines whether kss than a predetermined number of bits have 

been written in the dirQr RAM 124. 

If more than the predetermined nurnber of bits have been set. then the processor in stage S^^ 

pages of its memory 56 which have been dirtied, as indicated by the dirty bits read fi^ 

stage S43. to the memory 56 ofthe other processing set Qmtrol then passes back to stage S43. 

If, in stage S44. it is deterinined less than Ae predetermined nmiiber of Hts have been written 
RAM 1 24. then in Stage S45 the primary processor causes the bridge to inhibit DMA requests firom the devices 
connected to the D bus 22. Tlus coul4 for example, be achieved by chsaing the arbitration cnabfc bit 
the device slots, thereby denying access of the DMA devices to d« D bus 22. Altenuu^ 
142couldbecon£«medto ignore DMArequestsunderinstructionsfirantheprimaryproc^ Duringtheperiod 
in wMch DMA accesses are prevented the primary processor then makes a final copy pass ftom r^ 
memory 56 of the other processor fe those memory pages corresponding to the bits set inihe dirty 

to stage S47 Ae prirnary processor can issue a reset operation initiating a combmcd modt 

to stage S48, DMA accesses are once more pennitted. 

It win be twrndated ftat aMmu^ particuhir embodiments of die imrenticm h^ 
.nodifications/additions aiid/or sohstitutians may be inade wM^ 

defined in the appended daims. For aomiple; although in the specific descriptiim two processing seS 

it will be appreciated that the specifically described feamres may be modified to provide for three or more 

processing sets. 
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WHAT IS CLAIMED: 

1 A bridge for a multi-processor system, the bridge comprising: 

a first processor bus interfece for comiection to an I/O bus of a first processing set, Ae first processing 

set including memory; 

a second processor bus interface for connection to an I/O bus of a second processing set. the second 

processing set inchiding memory; 

a device bus interface for connection to a device bus; 

a bridge control mechanism configured to be operable to permit direct memory access to the memory 
o f d« processing sets by a deWce on the device bus. to arbitrate between the first and tte 
for access to the bridge in a first. spUt, mode, and to monitor lockstep operation of the first and second 
processing sets in a second, combined, mode; and 

a dirty RAM mechanism in ±s bridge for monitoring regions of processor set memory modified by 
direct memory access by the device on die device bus. 

2. nie bridge of claim 1. wherein the dirty RAM mechanism defines a dirty indicator for each of a 
plunJity of regions of processing set memory, a dirty indicator being set to a predrt^^ 
region of memory has been written to by a DMA access. 



3. 



Hie taidge of daim 2, wherein die dh^ indicator is a dirty bit 



4. TT« bridge of claim 2, wherein the processing sets are configured such that one of die processing sett 
is operable in the split mode as a primary processing set and to copy the coment of its memory to 4^ 

processing set 

5. Tl«lmdge of claim 3, wherein the primary processing set is operable at the end of a 

copy memory regions, which are identified in d« dirty RAM mechanism as having been written to by ^ 
d« corresponding dirty mdication being set. fitffli its memory to die memory of die ot^ 

6. Tie bridge of claim 4. wherein the bridge cmitrol mechanism comprises an arbiter configured to be 
operable in die split mode to arbitn^e for access to the bridge by tf« first and second proce«^ 

die device bus. 

7. TTie bridge of claim 6. wherein die bridge control mechanism is configured to be operable to respond 
K> a synchronization reset operation fiom die primary processing set, on completion of copying die content of 
die memory regions identified in die dirty RAM mechanism widi no fnidier regions having te^^ 

to transfer fh>m die split mode of operation to die combined mode of operation. 
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8. TT,e bridge of claim 7, whaein the dirty RAM nuxhmimro 
random access mcnuwy in dK bridge. 



9. 
set. 



TTie bridge of claim 6, wherein die content of the dirty RAM is cleaied on being lead by a processing 



10. The bridge of claim 1. comprising at least one fimher processor bus interfece for connection to an yO 
bus of a fiiTdier processing set 

„ A bridge for a multiimKessor system. d« bridge comprising means for interfa^ 
forafnst processing set.asecond I/O bus forasecondprocessing set. andadevice bus, means pen^ 
memory access to memory of the processing sets by a device on the device bus. means for aibitno^ 
&e fust and the secoml processing sets for access to the bridge in a first. spUt. mode, means 

lockstep operation of the first and second processing sets in a second, combined, m^ 

for monitoring regions of processor set memory modified by direct memory accesses by to 

device bus. 



12. A computer system comprising a first processing set having memory and an first DO bus. a 
p^xssing set having mnnory «Hi a second I/O bus. a device bus. at le«« one devk^ 
bridge, the bridge being cmmectedio the first I«b». the second W) bus and the device bu.^ 

a bridge control mechanism configmed to be operabte to pennft dirert memory access ID the m 

of fl« processing sets by the at least one device on the device bus. to arbitrate be*^ 

pHKessing sets for access to the bridge in a first, split, mode, and to monitor loc^ 

second processing sets in a second, combined, mode; and 

a dirty RAM mechanism in dK bridge for monitoring regimts of pnxxssor set memory modM^ 

direct memory accesses by die device on tf» device bus. 

13. A computer system according to daim 12, whemm each processing set comprises at least one 
processor, memory and a processing set I/O bus contK)ller. 

14. Ttecomputersystemofclaiml2,furthercomprismgatleastonefiirtherp^ 

15 A mediod of operating a muW-processor system comprising a first processing set having memory 
a fust I/O bus. a second processing set having memory and a second 1«) bus. a device bus having 
device competed thereto, and a bridge, the bridge being connected to die first yo bu^ 

the device bus. die mediod conqirising: 

permittu* direct memory access to d« memory of die pmcessing sets by are at 1«^ 

device bus; and 
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nionitoring. in a dirty RAM in the bridge, regions of processor set memory modified by direct mcmoiy 
access by the device on the device bus. 

16 A meflu)d of re-integrating a feulttolerammulti.im)ces8or system comiw^ 
having memory and an I«) bus. a second pnn^sing set having memory and an I/O bus. a 
leas, one device competed thereto, and a bridge, the bridge being com«c^ 

bus and the device bus, the method compiising: 

foIiowingalocksteperror.operatingthesysteminaspIitmodeinwUchoneofdM:p^^ is 

operable to copy its state to the od« imx^ssing set. during wMch spht mode direct memory acc« 

of the processing sets by the at least one device on the device bus is permitted and regions of pro^ 

memory vwitten to by the device are mariosd in a dirty RAM in the bridge; 

conductingamnnber of timesastep of copying areas of memory indicated in the dirty RAM as havi^^ 

been dirtied since the start of a previous copy step. 

17 TteoK^odofclaim 16. wherein, directmemory access is inhibited during a fi^ 

combined mode is initiated, m which combined mode lockstep opemtion of the first and second processing sets 
is monitored. 
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